CAMERA, IMAGING SYSTEM AND ILLUMINATION SENSOR 

This application is a continuation-in-part of U.S. 
Application No. 09/319,920, which was the National Stage 
5 of International Application No. PCT/ JP96/03683 , filed 
December 17, 1996. The international application was 
published in Japanese under PCT Article 21(2) . 

Technical Field of the Invention 

10 

The present invention relates to a camera, an 
imaging system and an illumination sensor which are 
capable of correcting colors of a captured image of an 
object to correctly display the colors of the image. 

15 

Description of Related Art 

Colors of objects are susceptible to illumination 
conditions, and it is therefore difficult to always 
20 display correct colors of images captured by a camera. 
Human eyes can correctly recognize actual colors of 
objects regardless of such conditions, which ability is 
known as color constancy. 

Existing video cameras do not comprise imaging 
25 devices having this feature. Attempts are being made to 
implement the color constancy in imaging systems having 
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such video cameras by performing complicated correction, 
e.g., by comparing color of a particular point with the 
surrounding color to make correction. However, these 
attempts are not practical, since they are limited to 
5 correction to special images or the image processing 
takes a long time. 

An object of the present invention is to provide an 
imaging system with good response which can correct 
colors of taken images of objects at practical speed to 
10 correctly display the colors of the images. 

Summary of the Invention 

15 A camera of the present invention comprising an 

imaging device for taking a color image and a lens for 
forming an image of an object on said imaging device, 
the imaging system comprising a reflection surface 
provided within a maximum field of view (Vm, Vm) formed 

2 0 by said lens and said imaging device, for diffuse- 
reflecting the image of said object to cause the image 
to impinge upon said imaging device through said lens, 
said imaging device having a direct image section for 
forming a direct image imaged as an object point on said 

2 5 object and an indirect image section for forming an 

indirect image of said object point obtained from said 
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reflection surface . 

The analysis by the inventor described later, 
revealed that a diffuse-reflected indirect image at the 
reflection surface provided within the maximum field of 
5 view, represents the brightness at the object point. 

Particularly, setting said reflection surface so 
that a direct image section for forming said direct image 
in said imaging device has a larger width than an 
indirect image section for forming said indirect image 

10 enables effective use of the maximum field of view of the 
imaging device. Moreover, as will be described later, it 
was confirmed that the color correction encounters no 
problem even when the width of the indirect image section 
is about 2 5% of the maximum field of view. 

15 It is preferred that the imaging system comprises a 

cover provided on the side on which light comes into said 
lens, for intercepting light from outside of said maximum 
field of view at least. While light outside the maximum 
field of view causes errors in color correction, the 

2 0 cover reduces the errors. 

The preferable way of correcting the color of the 
image is comparing the direct image and the indirect 
image . 

It is preferable that the white balance of the direct 
2 5 image were performed by using colors on the indirect 
image . 
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When designing the reflection surface, the direct 
image and the indirect image of said object may be 
similar figures with respect to the direction in which 
said direct image section and said indirect image section 
5 are arranged. In this case, it is possible to corrected 
color small objects like flowers more precisely to the 
details . 

The reflection surface may be so designed that the 
ratio of the numbers of corresponding pixels between the 

10 indirect image section for forming said indirect image 
and the direct image section for forming said direct 
image in the direction in which said direct image section 
and said indirect image section are arranged is constant. 
In this case, the algorithm for color correction can be 

15 simplified and the color correction can be achieved at 
very high speed. 

The reflection surface can be shaped according to 
the following equation: 

Xni = f (A— tan (2 a ) ) / (1+A • tan (2 a ) ) 

2 0 Where f represents focal length of the lens, A 

represents (X/"Z) , X represents horizontal-direction 
distance of the object point P from a horizontal- 
reference line Ch, Z represents vertical distance of the 
object point P from a vertical -reference line Cv, and a 

2 5 represents an angle formed between the reflection surface 
and a horizontal line parallel to the vertical- reference 
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line Cv. 

Experiments made by the inventor showed that 
forming said reflection surface with leather having oil 
or clear resin coating on its surface allows the color 
5 correction to be achieved very well. 

An imaging system of the invention is further 
comprising a video capture digitizing the image 
sequentially scanned along the scan lines n said camera 
and stores the data into a memory, the image having said 
10 direct image section and said indirect image section . 

It is preferable that the images were processed by 
dividing the colors on the direct image by colors on the 
indirect image in each color and the results by said 
dividing were multiplied with a common correction term 
15 (S) in three colors. 

This imaging system of this invention, preferably, 
further comprising assigning device (means) for assigning 
each main coordinates (Xmi, Ymi) in said direct image 
obtained as an object point on said object is imaged on 
20 said imaging device to corresponding sub-coordinates (Xni, 
Yni) in said indirect image of said object point (P) 
obtained from said reflection surface on said imaging 
device, and a color-correcting device (portion) for 
obtaining an image color-corrected on the basis of the 
25 following expressions: 

Dl (Xmi, Ymi) = (Rmi/Rni) "S, 
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(Gmi/Gni) ■ S, and 
(Bmi/Bni) ■ S. 

Where Dl, D2 , and D3 respectively represent R, G, B 
5 components of the corrected color image at said main 
coordinates (Xmi, Ymi) , Rmi, Gmi , and Bmi respectively 
represent R, G, B components in a direct image pixel (Pm) 
at said main coordinates (Xmi, Ymi), Rni, Gni , and Bni 
respectively represent R, G, B components in an indirect 

10 image pixel (Pn) at said sub-coordinates (Xni, Yni) , and 
S represents a correction term. 

The analysis by the inventor described later, 
revealed that a diffuse-reflected indirect image at the 
reflection surface provided within the maximum field of 

15 view, represents the brightness at the object point. 

Accordingly, dividing Rmi , Gmi, and Bmi respectively by 
Rni, Gni, and Bni representing the brightness eliminates 
errors due to effects of illumination. This was 
confirmed by experiments carried out by the inventor. 

20 The correction term S prevents the output resulting from 
the division of Rmi, Gmi, Bmi by Rni, Gni, Bni from 
exceeding the limit of device scale width and becoming 
saturated. 

An illumination sensor of the invention further 
2 5 comprising an imaging device for taking a color image and 
a lens for collecting the illumination, the illumination 



D2 (Xmi, Ymi) = 
D3 (Xmi, Ymi) = 
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color sensor comprising a reflection surface provided 
within a maximum field of view (Vm, Vm) formed by said 
lens and said imaging device , for diffuse-reflecting the 
image of said object to cause the image to impinge upon 
5 said imaging device through said lens , sensing the 

illumination color from the indirect image of said object 
point obtained from said reflection surface on said 
imaging device. 

The present invention can be implemented by 

10 installing software for realizing the assigning device 
stored in a storage medium into a common personal 
computer and attaching the cover having the reflection 
surface to a common video camera. 

As stated above, the features of the present 

15 invention provides an imaging system with good response 

which can correct colors of a taken image of an object at 
practical speed by comparing an indirect image from the 
reflection surface and a direct image so as to correctly 
display the colors of the object. 

2 0 The present invention will become more apparent 

from the following detailed description of the 
embodiments and examples of the present invention. The 
reference characters in claims are attached just for 
convenience to clearly show correspondence with the 

2 5 drawings, which are not intended to limit the present 

invention to the configurations shown in the accompanying 



7 



drawings . 

Description of the Drawings 

5 Fig.l is an explanation diagram showing the 

relation among an object Ob, reflection surface, lens, 
and CCD device for explaining the principle of the 
pre sent invent ion ; 

Fig. 2(a) is a diagram showing the first 
10 convolution on the reflection surface due to diffuse 

reflection pattern, and Fig. 2(b) is a diagram showing 
the second convolution on the CCD device due to 
defocusing on the indirect image section Fn; 

Fig. 3 shows the relation between a direct image Im 
15 and an indirect image In on the total image plane F; Fig. 
3(a) in a nonlinear mapping, Fig. 3 (b) in a linear 
mapping, and Fig. 3(c) with spot light illumination; 

Fig. 4 shows assignment between direct image pixel 
groups Pm and indirect image pixels Pn; Fig. 4(a) in the 
20 nonlinear mapping, and Fig. 4(b) in the linear mapping. 

Fig. 5 (a) is a graph showing the relation between 
varying horizontal location of the object point P and the 
optimum reflection surface angle a, and Fig. 5(b) is a 
graph showing the relation between the depth of the 
25 object point P and the viewing error angle \J/; 

Fig. 6 (a) is a graph showing changes before and 
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after color correction in the relation between the 
illumination intensity and brightness, and Fig. 6(b) is a 
graph showing changes before and after color correction 
in the relation between the illumination intensity and x- 
5 chromaticity coordinate; 

Fig. 7 (a) is a perspective view of a camera fitted 
with a guard cover, and Fig. 7(b) is a transverse 
sectional view showing the guard cover; 

Fig. 8 is a logical block diagram showing an imaging 
10 system according to the present invention; 

Fig. 9 is a diagram showing another embodiment of 
the present invention, which corresponds to Fig.l; and 

Fig. 10 is a block diagram of another embodiment of 
the invention, which comprise not only a camera but also 
15 an illumination sensor. 

Detailed Description of the Invention 

First, the principle of the present invention will 
2 0 be described referring to Figs.l to 5. 

The example of Fig.l shows a simplified geometric 
model of optical paths. Now we consider the positional 
relation between an object point P on a general object Ob 
and a reflection reference point N, or a particular point 
25 on a reflection surface (nose surface) 18. The object 

point P on the object Ob is focused as a direct image on 
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a direct image section Fm of a CCD device 23 through "O" 
in the lens 21 of the camera 20. The image of the object 
point P on the object Ob is also diffuse-reflected at the 
reflection surface 18 of the reflector 15 and passes 
5 through the lens 21 to impinge upon an indirect image 

portion Fn of the CCD device 23 to form an indirect image. 
While the indirect image section Fn is not focused 
because the diffuse reflection at the reflection surface 
18 and the reflection surface 18 are out of the focus of 

10 the lens 21, it is assumed here for simplicity that the 
reflection surface 18 causes mirror reflection, and the 
centers of the optical paths are shown as segments PN and 
NO for convenience . 

The range surrounded by a pair of maximum field 

15 lines (planes) Vm, Vm on the total image plane F of the 
CCD device 23 is the range in which the lens 21 can form 
an image on the CCD device 23, which corresponds to the 
maximum field of view. Needless to say, the maximum 
field of view extends in the direction perpendicular to 

2 0 the paper of Fig.l. In the total image plane F 

corresponding to this range, the regional indirect image 
section Fn surrounded by the maximum field line Vm 
extending from upper left to lower right and the boundary 
field line Vn connecting the reflection surface top 18a 

25 of the reflection surface 18 and the u O" in the lens 21 
is the range in which the indirect image is formed. The 
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remaining regional direct image section Fm is the range 
in which the direct image is formed. 

The horizontal -reference line Ch in Fig.l is a 
reference axis passing through the center of the lens 21 
5 to show the zero point with respect to the horizontal 

direction and the direction of the thickness of the paper, 
and the vertical -reference line Cv passing through the 
imaging plane of the CCD device 23 is a reference axis 
showing the reference point with respect to the vertical 

10 direction. The image coordinates are represented by (X, 
Y, Z) system coordinates. The characters X, Xn, Xmi, and 
Xni in the drawing show the horizontal distances between 
the horizontal-reference line Ch and the object point P, 
reflection reference point N, direct image Im of the 

15 object point P on the direct image section Fm, and 

indirect image In of the object point P on the indirect 
image section Fn, respectively. Similarly, the 
horizontal distances between these points and the 
horizontal-reference line Ch in the direction 

20 perpendicular to the paper of Fig.l are shown by the 

characters Y, Yn, Ymi and Yni . The characters Z and Zn 
in the drawing indicate the vertical distances between 
the vertical -reference line Cv and the object point P and 
the reflection reference point N, respectively. In other 

2 5 words, the distance Zn designates the depth of the 
reflection reference point N, and then the vertical 



11 



direction distance between the object point P and the 
reflection reference point N is given as Z — Zn. 

Light from an illuminant hits the surface of an 
object and is then reflected in a form dependent on the 
optical characteristics of the surface. The reflection 
seen by the camera 20, 1(A), is given by the following 
expression. 

I (A) =E (A) ■ p (A) 

(1 ) 

Where E (X) is the spectral power distribution of 
the illumination, p(A) is the surface reflectance of the 
object, and A is the wave length. The reflection 1(A) is 
then decomposed into three colors R, G, and B. The 
reflection surface 18 reflects the illumination at a 
horizontally reduced resolution, as compared with that of 
the direct image Im, and obtaining the indirect image In 
means a measure of the illumination in this way. 

The specular-reflected ray at the reflection 
surface 18 is surrounded by a distribution of diffuse 
rays. It affects the ray reaching the indirect image 
section Fn on the CCD device 23 by different weights. 
For example, in Fig. 2 (a), the rays incident along the 
optical axes SI and S2 have diffuse reflection intensity 
distributions approximated to the Gaussian distributions 
Gl and G2 having their respective peaks on the optical 
axes of specular reflections, SI and S2 . The ray on a 



12 



particular optical axis Sn toward the CCD device 23 
affects, with values of intensity of DRC1 and DRC2 , the 
rays reaching the CCD device 2 3 as the indirect image 
section Fn. With this approximation, the ray C reflected 
5 from a point on the nose can be expressed as follows: 
C(X,Y) = JjEo(X,Y)-po(X,Y)-Bl(X,Y)dXdY 

(2) 

Where the letter w o" refers to a point on the 
object Ob. The ray C represents a weighted summation of 

10 all rays illuminating point N on the nose surface from 
the scenery points. The weighting factor changes with 
the changing angle of incidence and the roughness of the 
nose surface. The blurring factor Bl depends on the 
optical characteristics and roughness of the reflection 

15 surface 18. 

As the reflection surface 18 is seen out of the 
focus of the lens 21, every ray reflected from the 
reflection surface 18 will be projected as a circle. The 
intensity distribution is approximated to vary according 

2 0 to Gaussian function across the diameter of the blur 
circle as shown in Fig. 2(b) . Thus every pixel in the 
indirect image section Fn of the CCD device 23 receives a 
weighted summation of a circular window. The size of 
this window depends on the blurring factor B2 , which in 

25 turn depends on the focal length and the depth of the 
reflection reference point N from the lens 21 of the 
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camera 20. 

Cn i = J* J B 2 (Xni, Yni) • C (Xni, Yni) dXni-dYni 

(3) 

Where the letter ni refers to pixel of the indirect 
5 image In and the sub- coordinates (Xni, Yni) refer to the 
coordinates of the indirect image on the CCD device 23. 

Combining the two expressions containing the two 
kinds of blurring factors Bl and B2 results in two 
operations of spatial blurring. One is achieved on the 

10 reflection surface 18, and the other is achieved by 
defocusing on the CCD device 23 since the reflection 
surface 18 is imaged out of the focus of the lens 21. 
The blurring process is performed in two separately- 
controlled layers. We assume that the successive 

15 convolutions by combining the two expressions containing 
the two blurring factors represents the illumination 
color at the object point P. That is to say, we consider 
that the indirect image section Fn obtained on the CCD 
device 23 by reflection on the reflection surface 18, 

20 represents the illumination color at the object point P 
or illumination in the vicinity thereof. 

Accordingly, the color intensity signals Dl, D2 and 
D3 obtained by calculation shown in the following 
expressions (4) represent corrected colors of the object 

25 point P on the object Ob. This is due to the fact that 
dividing Rmi , Gmi , Bmi at each main coordinate 



representing the color at the object point P itself by 
Rni, Gni , Bni at the sub-coordinates representing the 
color at the object point P removes the effects of 
illumination color etc. at the object point P. 
5 D 1 (Xmi, Ymi) = (Rmi/Rn i ) • S, 

D 2 (Xmi, Ymi) = (Gm i /G n i ) ■ S, 
D 3 (Xmi, Ymi) = (Bmi/Bn i)'S 

(4) 

Where the letter m represents the direct image Im, 

10 n represents the indirect image In from the reflection 
surface 18, and i represents an image on the CCD device 
23. The characters Dl , D2 and D3 respectively represent 
R, G, and B components of the color-corrected image at 
the main coordinates (Xmi, Ymi) , Rmi, Gmi, Bmi 

15 respectively represent R, G, B components in a direct 

image pixel (Pm) at the main coordinates (Xmi, Ymi), and 
Rni, Gni, Bni respectively represent R, G, B components 
in an indirect image pixel (Pn) at the sub-coordinates 
(Xni, Yni) . The main coordinates (Xmi, Ymi) stand for 

2 0 coordinates of the direct image obtained when the object 
point P is focused on the imaging device 23 and the sub- 
coordinates (Xni, Yni) stand for coordinates of the 
indirect image of the object point P obtained by the 
reflection surface 18 on the imaging device 23. The 

25 factor S adjusts the absolute values so that the values 
Dl to D3 will not be saturated. 
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The role of the reflection surface 18 as a sensor 
for detecting spatial illumination can be confirmed by a 
simple experiment. When a strong spot light is directed 
to a white wall, the imaging system 1 of the invention 
5 takes the image shown in Fig. 3(c) . The direct image Im 
of the spot appears as an image like a white circle on 
the left side of the boundary DL, and its indirect image 
In is projected at a reduced horizontal resolution as an 
ellipse with a surrounding flare. The reflection at the 

10 reflection surface 18 represents the illuminant color. 
The color of illumination can be changed by using color 
filters with an incandescent lamp. The narrow band light 
was projected on a white wall, and the R, G and B values 
were measured for corresponding patches in the direct 

15 image Im and the indirect image In. The ratios of the 

color intensity signals (Dl, D2 , D3) were almost constant 
when the color of illumination was varied. 

Next, the positional relation between the 
reflection surface 18 and the camera 20 will be described. 

20 The reflection surface 18 and a horizontal line 

parallel to the vertical -reference line Cv forms an angle 
a. The reflected ray from the reflection surface 18 
represented by the line NO and the horizontal line forms 
an angle and the line NO and the reflection surface 18 

25 forms an angle (3. The line NO and a perpendicular line 

to the reflection surface 18 forms an angle 0. Since the 



line NO indicates specular reflection of the line PN 
indicating the incident light at the reflection surface 
18, the line PN and a perpendicular line to the 
reflection surface 18 also forms the angle 0. The 
5 character f denotes the focal length of the lens 21 of 
the camera 20. The angle formed between the line PN and 
a horizontal line is designated as x, the object point 
horizontal location angle between the line PO and a 
horizontal line as <p, and the viewing error angle between 
10 the line PO and the line PN as ^ . 

For the object point P, ^ = 0 — x (5) . 

For Z PNO , x 29. 
For the vertically opposite angle about the 
reflection reference point N, a = £ + fi holds. 
15 From the relation about the perpendicular line to 

the reflection surface 18 around the reflection reference 
point N, fi =n /2 — 6 . 

From the above two expressions around the 
reflection reference point N, <£ = ar — 13= a + 0— 7r/2 holds, 
2 0 and further, = — <x + tx:/'2 holds. 

When the expressions above are rearranged, the 
following equation holds: 

^ = 0-x=0-(2 9- ^)=0 + ^- 2 0 = 0 + ^- 2 -a + Tt/2) 

25 

(6) 
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The angle a of the reflection surface 18 can be 
calculated using the equation above. The object point 
horizontal location angle cp can be obtained using the 
equation below. 
5 0=tan _1 ((Z-f)/x) 
(7) 

The angle £ is an index indicating the horizontal 
direction coordinate of the reflection reference point N 
on the reflection surface 18 or the indirect image In, 
10 which can be obtained by 

£ = t a n" 1 ( f /X n i ) 

(8) 

The optimum angle a of the reflection surface 18 
with the changing horizontal coordinate of the object 

15 point P is shown in Fig. 5 (a). The angle a was calculated 
by setting the viewing error angle f to a small value of 
2 degrees. Other angles were represented by their 
average magnitudes. In Fig. 5 (a), the object point 
horizontal location angle cp is shown on the abscissa and 

2 0 the angle a of the reflection surface 18 is shown on the 
ordinate. It is preferred that the angle a of the 
reflection surface 18 be appropriately decreased when the 
object point horizontal location angle cp increases, so as 
to keep the viewing error angle f to a small and almost 

25 constant value. 

As shown in Figs . 1 , 3 and 4, each image line 

18 



consists of the direct image section Fm for taking the 
direct image Im and the indirect image section Fn 
separated by the boundary DL for taking the indirect 
image In. The boundary DL separating the direct image 
5 section Fm and the indirect image section Fn corresponds 
to the reflection surface top 18a of the reflection 
surface 18. Mapping in this invention is defined as 
assigning indirect image pixels Pn forming the indirect 
image section Fn to direct image pixel groups Pm in the 

10 direct image section Fm. Mapping becomes difficult when 
the object Ob is located near the camera 20, for the 
angle \|f is a measure of the viewing error. The angle i|/ 
is required to be as small as possible to minimize the 
view difference between the direct image Im and the 

15 indirect image In. When the angle i|/ is large, the' object 
Ob will be seen in the direct image section Fm, but not 
be visible in the indirect image section Fn, or vice 
versa. The angle i|j can be expressed in terms of 
coordinates of the reflection surface 18. Referring to 

20 the geometry of Fig.l, the following expression can be 
derived . 

t an(x) = (Z-Zn)/(Xn-X) 

(9) 

Obtaining tangents on both sides in Eq. (5) provides 
25 the following equation. 

tan0=(tan<£ — tan(x))/(1+tan0- tan 
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(x)) (10) 

From Eqs.(9) and (10) above, the following equation 
can be obtained. 

t an0=(Xn(Z-f)+Zn-X + X- f- 2X-Z)/ 
5 (X n -X + Z n ( f -Z) +Z (Z- f ) -X 2 ) 

(1 1 ) 

Eq. (11) represents the dependency of the angle on 
both of the object point P (X, Z) and the reflection 
reference point N (Xn, Zn) on the reflection surface 18 . 
10 When X is set to be equal to zero in Eq. (11) , the value 

of tangent angle at the camera optical axis points can be 
obtained as shown by Eq. (12) below. 

t an*=(Xn(Z-f)/(Zn(f-Z)+Z(Z-f)) 

(1 2) 

15 An increase in the reference point horizontal 

distance Xn, or the horizontal distance between the 
reflection reference point N and the horizontal -reference 
line Ch, will result in a subsequent increase in the 
viewing error Therefore it is preferable to place the 

20 reflection surface 18 as horizontally close to the 

horizontal -reference line Ch as possible. The error 
angle \|j increases with an increase in the depth Zn of the 
reflection reference point N. Thus the depth Zn of the 
reflection reference point N from the lens 21 of the 

25 camera 20 should be as small as possible. 

Fig. 5(b) shows the dependence of the viewing error 
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angle on the object distance Z. An increase in the 
distance Z decreases the viewing error angle \|f. The 
error angle \|/ will be at considerable values for objects 
Ob placed near, but it is less than 2 degrees at 
5 distances of 4 0 cm or longer. The viewing problem is not 
serious unless the lighting is made in stripes with fine 
resolution. For normal lighting conditions, the 
illuminance does not exhibit high frequency changes. The 
error angle ijr increases as the reference point horizontal 

10 distance Xn from the reflection reference point N 

increases. This effect is minimized if the angle aof the 
reflection surface 18 changes according to the trend 
shown in Fig. 5 (a) . 

From Eqs . (5) and (6) above, x = 0 — z// = n + £ — 2a, 

15 and further, obtaining tangents of these equations 
provides the following equation. 

t a n ( x ) = t a n ( tt + £ — 2 a ) = t a n ( £ — 2 or) 

(1 3) 

When Eqs. (7) and (8) are substituted in the 
2 0 equation above, the following equation is obtained. 
(Z-Z n )/(X-X n ) 
= ( ( f - X n i-tan(2a))/(Xn i + f- tan(2a)) 

(1 4) 

With (X/Z)=A, Eq. (14) above can be expanded and 
25 rearranged to present the following equation. 
Xn i/f 
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= ((A-t a n (2a))-(Xn/Z-(Zn/Z)' t a n (2a))) 
/ ( ( 1 + A ■ t a n (2 a) ) — ( (X n/Z) • t an(2ar)+Zn/ 
Z) ) (15) 

Further, when Z>>Zn, and X>>Xn, the latter terms 
5 in the numerator and denominator are then equal to zero, 
and the following equation holds. 

Xn i=f (A-tan(2a))/(1+A-tan(2o)) 

(1 6) 

This equation describes the mapping on the 

10 horizontal coordinate between the direct image Im and the 
indirect image In of the object Ob in the same scan line 
SL. Xni designating the coordinate of a point on the 
indirect image section Fn corresponding to one object 
point P on the object Ob does not clearly depend on the 

15 value of the distance, but rather depends on the ratio 
A=(X/Z) . This can be explained by considering the 
omission of Zn in the equation. When it is assumed that 
the object Ob is sufficiently distant from the position 
of the camera 20, then the angle will be very small. 

20 In this case, if the object point P moves along OP, the 

reflection at the reflection surface 18 on the segment PN 
slightly changes. As shown by Eq. (16), the mapping is 
directly related to the determination of the profile of 
the reflection surface 18. 

25 Figs. 3 (a) and 4(a) show a method of nonlinear 

mapping, and Figs. 3(b) and 4(b) show a method of linear 
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mapping. The mapping can be selected between the 
nonlinear and linear relations by defining Xni , A= (X/Z) 
and the angle a of each small part of the reflection 
surface in Eq. (16) in the positional relation between the 
5 direct image section Fm and the indirect image section Fn. 
Fig. 4 shows the correspondence between a direct image 
pixel group Pm on the direct image section Fm and an 
indirect image pixel Pn on the indirect image section Fn 
on one scan line SL, where the arrows show the direct 

10 image pixel groups Pm and the indirect image pixels Pn 
which correspond to each other when they are obliquely 
shifted. While the direction of mapping in the direct 
image section Fm is directed as shown by the arrow Mm, 
the mapping direction in the indirect image section Fn is 

15 directed as shown by the reversely-directed arrow Mn. 

Usually, the boundary DL between the direct image section 
Fm and the indirect image section Fn is perpendicular to 
the lower edge of the total image plane F. 

In the nonlinear mapping shown in Figs. 3 (a) and 

20 4(a), the indirect image pixels Pn are assigned to 

corresponding direct image pixel groups Pm composed of 
different numbers of pixels. In this mapping, the 
dimensions of the corresponding portions in the direct 
image Im and the indirect image In are assigned so that a 

25 /d = b/c. That is to say, they are so assigned that the 
direct image Im and the indirect image In of the object 
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Ob are similar figures with respect to the direction in 
which the direct image section Fm and the indirect image 
section Fn are arranged. This mapping is suitable for 
precisely color-correcting small parts in an image, e.g., 
5 when taking pictures of small objects like flowers. 

In the linear mapping shown in Figs. 3(b) and 4(b), 
they are so assigned that the ratio of the numbers of 
corresponding pixels between the indirect image section 
Fn and the direct image section Fm, (Pm/Pn) , is constant 

10 in the direction in which the direct image section Fm and 
the indirect image section Fn are arranged. In this 
mapping, the dimensions of corresponding parts in the 
direct image Im and the indirect image In are so assigned 
that a/d and b/c are not uniform. That is to say, the 

15 direct image Im and the indirect image In cannot be 

similar, and the parts in the direct image section Fm are 
color-corrected at uniform resolution. This mapping 
enables high speed image processing, which provides 
color-corrected image almost on a real-time basis. The 

2 0 assigning device for performing the assignment can be 

implemented by using a personal computer 30, as will be 
described later. 

The entirety of the reflection surface 18 is not 
linear as shown in Fig.l, but individual small parts have 

25 different surface angles a, the entirety of which is 

formed in a curved shape. The reflection surface 18 is 
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drawn in a linear shape in Fig.l only for convenience in 
description . 

When graphically designing the reflection surface 
18, first, the angle a of a small part on the reflection 
5 surface at the reflection surface top 18a is determined 
such that the visible image vertical extreme lines can be 
projected on the indirect image section Fn on the right 
side of the boundary DL on the CCD device. The angle a 
of each small part on the reflection surface was 

10 determined on the basis of the requirement shown in 

Fig. 5 (a) . The Length projected from a direct image and 
the corresponding length of small part on the reflection 
surface 18 were graphically measured at a depth of one 
meter. In this case, the difference in depth did not 

15 cause considerable errors as estimated numerically from 
Eq. (16) . That is to say, it can be said that a mapping 
equation is fitted to the graphical measurements of 
pixels between the direct image section Fm and the 
indirect image section Fn. 

2 0 When numerically designing the reflection surface 

18, first, the coordinates of the reflection surface top 
18a, (Xo, Zo) , are obtained from the boundary of the 
light reaching the camera 20. When using the linear 
mapping, the above-described A = (X/Z) and M = (Xni/'f) 

2 5 are obtained from the correspondence between the indirect 
image section Fn and the direct image section Fm, and the 
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angle a of the small part of the reflection surface at 
that coordinates is determined by using Eq. (16) . Next, 
by using the following equations (17) -and (18) , 
coordinates of a part separated from the coordinates (Xo, 
5 Zo) by a small distance are obtained. The characters 

"n" and "n-l" in the following two equations indicate 
^ relation between an (n — l)th small part closer to the 

reflection surface top 18a, when the reflection surface 
18 is divided into small parts, and an nth small part 
10 location on the side closer to the reflection surface end 
18b away from the top. Further, the angle a at the newly 
obtained coordinates (Xn, Zn) is obtained from Eq. (16) 
above. This process is sequentially repeated to 
determine the curved plane of the reflection surface 18. 
15 Z n =(Z„. r tana n . 1 (M n -f n -X n . 1 )) 
/(I -M n - t a n (<*„.,) ) 

(1 7) 

X n = X n _ n + (Z n . r Z n ) /t a n,^ 

(1 8) 

2 0 Next, a structure of an imaging system according to 

the present invention will be described referring to 

Figs . 1 , 3 , 4 , 7 and 8 . 

Fig. 7 shows a specific structure around the camera 

20, where a guard cover 10 having a reflector 15 is 
25 attached to the front side of the camera 20. The guard 

cover 10 has a body 11 shaped in a prismoid and a fitting 
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part 12 to be fitted on the camera 20. The body 11 of 
the guard cover 10 prevents invasion of light into the 
camera 2 0 from outside of the range surrounded by the 
pair of maximum field lines (planes) Vm, Vm shown in 
5 Fig.l. Intercepting the light from outside of the range 
surrounded by the pair of maximum field lines Vm, Vm is 
desired because it gives error to the light from the 
reflection surface 18 for correcting the image. 

The reflector 15 is attached on one of the inner 

10 sides of the body 11, which comprises a base 16 for 

defining the surface shape of the reflection surface 18 
and a skin 17 stuck on the surface of the base 16. The 
surface of the skin 17 on the reflection surface 18 side 
is matte and black- or gray-colored to dif fuse-reflect 

15 light, on which oil is applied to form film. 

Fig. 8 is a logical block diagram showing the 
imaging system 1; the imaging system 1 includes, as main 
members, the guard cover 10, camera 20, personal computer 
30, monitor device 41, and color printer 42. The image 

2 0 captured through the lens 21 of the camera 2 0 is formed 
on the CCD device 23 with its quantity of light adjusted 
through the diaphragm 22 . The output from the CCD device 
23 is captured into the video capture 31 in the personal 
computer 30 and is also given to the frame integrator 24, 

25 which obtains the quantity of light of the taken image to 
control the quantity of aperture of the diaphragm 2 2 with 
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the aperture motor 25 so that the output of the CCD 
device 23 will not be saturated. 

The personal computer 3 0 is a common product, which 
is constructed by installing software in storage means 
like a hard disk, RAM, etc. to implement various 
functions of the timer 32, color application circuitry 37, 
etc. described later. This software can be distributed 
in a form stored in a storage medium such as a CD-ROM, 
flexible disk, etc. The video capture 31 digitizes the 
image sequentially scanned along the scan lines SL in the 
camera 2 0 and stores the data into a memory. The timer 
32 functions as trigger for determining the position of 
the boundary DL for separating the direct image section 
Fm and the indirect image section Fn in the total image 
stored in the memory. In this embodiment, the direct 
image section Fm in the total image contains 24 0 pixels 
and the indirect image section Fn contains 80 pixels. 
The mapper 3 3 maps individual indirect image pixels Pn 
contained in the indirect image section Fn, 80 per one 
scan, to corresponding direct image pixel groups Pm in 
the direct image section Fm. This mapping is performed 
in a nonlinear or linear manner according to Eq. (16) as 
exp 1 a i ne d above . 

The color corrector 34 obtains Dl, D2 , and D3 
according to Eq. (4) and the maximum selector 3 5 obtains 
the maximums value of these values in the full image. 
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The level at which the maximums value is not saturated 
corresponds to the appropriate value of the correction 
term S as the factor in Eq. (4) , and the scaler 36 
determines the appropriate value of the correction term S 
5 in the color corrector 34, and the values of the outputs 
Dl, D2 and D3 are corrected. For example, with a 8 -bit 
computer, the scale width in information processing is 
256 and a scale width of about 85 is assigned to each of 
R, G, B, and therefore the correction term S is set so 

10 that the maximum value of the scale width for Dl , D2 , and 
D3 is 85 or smaller. Larger scale widths can be assigned 
with 16- or 32-bit computers, so as to represent colors 
at finer tones. 

The color application circuitry 37 serves as means 

15 for storing, reproducing, editing, etc. the color- 
corrected image, which is implemented by driving software 
stored in a hard disk or the like by a CPU or other 
hardware. The image reproduced by the color application 
circuitry 3 7 is displayed as color moving picture, for 

2 0 example, in the monitor device 41 through the video 

accelerator 38, and is also printed in colors as still 
picture through the I/O port 39 and the color printer 42. 

To verify the invention, a SONY XC-711 (trademark) 
color video camera was used as the camera 2 0 and a 12.5 

2 5 mm focal length COSMICAR C-mount lens (trademark) was 

used as the lens 21. The color parameters were measured 
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using a MINOLTA chroma meter module CS-100 (trademark) . 
The prototype configuration of the imaging system 1 was 
used to obtain still images to obtain experimental data. 
While the skin 17 was made of gray matte paper to cause 
5 diffusion reflection, the use of leather having a coat of 
oil provided better results. The width ratio of the 
indirect image section Fn to the total image plane F was 
limited to a maximum of 25%. The processing time on the 
personal computer 30 using Pentium (trademark) with 120 
10 MHz operation clock was 0.55 second for a 320x220 pixel 
image . 

The applicability of the imaging system 1 which 
corrects surface color can be confirmed by studying the 
effects of illumination intensity on color quality. The 

15 inventor carried out experiments with some pieces of full 
color images subject to combined daylight and fluorescent 
illumination. The images were processed by dividing the 
colors on the direct image Im by colors on the indirect 
image In. The image colors were improved, and dark 

2 0 images got brighter with observable details while strong 
illumination pictures got darker. Dark images below 100 
lx were noisy even after processed by the method using 
the reflection surface 18 . 

A separate experiment was carried out to detect the 

25 quantitative quality of color correction provided by this 
imaging system 1. A red color patch was set on a camera 
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plane and the color of the patch was compared at 
different illumination intensities. The effect of 
lighting intensity on the brightness of the red color 
patch before and after correction is shown in Fig. 6 (a). 
5 The abscissa shows the illumination intensity and the 

ordinate shows the brightness of the red color patch. As 
shown in the curve " Bef orecorrection" , an increase in 
scene illumination intensity usually results in an 
increase in brightness of the color patch in the image. 

10 As shown in the curve " by After correction" , the 
brightness of the patch after corrected is almost 
constant and stable even when the illumination intensity 
is changed. The effect of illumination intensity on the 
x- and y-chromaticity coordinates based on CIE 1931 

15 standard is shown in Fig. 6(b). As shown in the curve by 
Before correction in Fig. 6(b), the x-chromaticity 
coordinate of the red color patch shown on the ordinate 
increases as the illumination intensity shown on the 
abscissa increases. This implies a hue distortion of the 

20 original color at different lighting intensities. The x- 
chromaticity coordinate in the corrected images decreases 
slightly as the illumination intensity increases. While, 
as shown in Figs. 6 (a) and (b) , the values of brightness 
and x-chromaticity coordinate at 100 lx corresponding to 

25 the lowest illumination intensity are always different 

from those at larger intensity points, it is possible to 
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maintain the constancy of illuminance and hue at lower 
illumination intensities by changing the conditions for 
setting the reflection surface. 

The correction to image colors by the imaging 
5 system 1 using the reflection surface 18 eliminated 

distorted original colors of images. The corrected color 
histograms of one image under different light intensities 
were all similar. This shows that the lighting intensity 
has no global effect. As shown in Figs. 6 (a) and (b) , the 

10 color parameters before and after color correction show 

that the color brightness and hue vary only slightly when 
the illumination intensity varies in a certain range. 

Then, other possible embodiments of the invention 
will be described. 

15 Although the total image plane F in the CCD device 

23 is plane-shaped in the above -described embodiment, it 
is logically possible to use a CCD device 23 having its 
total image plane F shaped in a curved surface around the 
point O in the lens 21, for example. In this case, 

2 0 Eq. (15) shown above can be replaced by the equation below, 
t a n (2 a) 

= ( A - t anf + 1- ((Zn/Z) + (Xn/Z)'t a n £) ) 
/(I - A- t a n £ + ( (Z n/Z ) ■ t a n £ — (X n/Z) ) ) 

(1 9) 

25 Further, when Z>>Zn, and X>>Xn, the latter terms 

in the numerator and denominator in the equation above 
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are then equal to zero and the following equation for 
replacing Eq. (16) holds, 
t a n ( 2 a ) 
= ( A - t an£ + 1)/(1— A-t a n £ ) 

5 (20) 

The curved surface of the reflection surface 18 can 
be designed on the basis of this equation (20) in place 
of the equation (16) shown before. 

Although the reflection surface 18 is made of black 

10 leather having oil coating thereon in the above-described 
embodiment, the reflection surface may be made of other 
material having matte gray surface, for example, as long 
as it causes diffuse reflection of light at appropriate 
intensity or be made of meatl foil or layer like aluminum, 

15 and the surface may have varnished resin coating. 

Although using the CCD device 2 2 for the imaging 
device in the above-described embodiment, it may be 
possible to use some kind of devices such as CMOS device 
for the imaging device. 

2 0 In the embodiment above, in mapping, one indirect 

image pixel Pn is assigned to a direct image pixel group 
Pm composed of a plurality of direct image pixels to show 
a practical example. However, one indirect image pixel 
Pn may be assigned to one direct image pixel Pm when the 

25 indirect image section Fn and the direct image section Fm 
have equal width, and, it is also logically possible to 
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assign an indirect image pixel group Pn composed of a 
plurality of indirect image pixels to one direct image 
pixel Pm when the indirect image section Fn has larger 
width than the direct image section Fm. 
5 The average value of an indirect image pixel group 

Pn can represents the illumination color of the main 
image scope, so the main image color or white balance 
color can be corrected by the average value without 
mapping. In this case, the area of the indirect image 

10 section Fn can be smaller than the above. 

Finally, yet another embodiment of the present 
invention will be explained while referring to Fig. 10. 
In this embodiment, in addition to the main camera 20', 
an illumination color sensor 50 is provided. The average 

15 value of the multiple pixels of the above-mentioned 

indirect image collected at CCD 22 via the reflection 
surface 18 and the lens 21 is computed with the 
processing section 60, and the illumination colors taken 
with the main camera 20' are determined with this sensor 

20 50. These illumination colors are output as OP1 for each 
separate RGB components. Meanwhile, the main image of 
the object is taken with the main camera 2 0 ' via the 
lens 21 9 and the CCD 22' . The output of this main image 
is connected to amplifiers 34a, 34b, and 34c provided for 

2 5 each respective color of the three colors RGB. Output 
RGBs of the procession section 60 are respectively 
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connected to each amplifier, white balance adjustment is 
performed referring the illumination colors. However, for 
the white .balance adjustment, it is not always necessary 
to control all three colors; as shown in reference 
5 numerals 34a and 34b of the drawing, from among the three 
colors, if at least two colors are controlled, then such 
is sufficient. The same applies to the above-described 
software processing, too. However, it is desirous to 
perform control for all three colors in order to get 
10 maximum use of the dynamic range of the CCD. The 

corrected results of the main camera 20' are output from 
OP2 . It is noted that the OP1 stated above may also be 
used for color determinations of other illumination 
colors . 

15 The above-described imaging system (apparatus) can 

be applied to adjusting the white balance of a video 
cameras for taking moving pictures and digital cameras 
for taking still pictures, etc. The imaging system can 
also be preferably applied not only to stereo range 

20 finders on a color basis but also to the illumination 

color sensor. Current stereo range finders are designed 
to detect characteristic points on every scan line on the 
basis of change of color code. The characteristic points 
are compared between the right and left stereo images, 

25 and the correspondence is determined when the color codes 
are similar. Stabilization of the color codes is a major 
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advantage of the color constancy of the present invention, 
and application of this imaging system to the stereo 
range finders enhances the stereo matching reliability. 
The disclosure of U.S. Patent Application No. 
5 09/319,920 filed on June 17, 1999 including specification, 
drawings and claims is incorporated herein by reference 
in its entirety. 

Although only some exemplary embodiments of this 
invention have been described in detail above, those 

10 skilled in the art will readily appreciated that many 

modifications are possible in the exemplary embodiments 
without materially departing from the novel teachings and 
advantages of this invention. Accordingly, all such 
modifications are intended to be included within the 

15 scope of this invention. 
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